Minimizing Cumulative Error in Discourse Context

Cumulative error limits the usefulness of context in applications utilizing contextual information. It is especially a problem in spontaneous speech systems where unexpected input, out-of-domain utterances and missing information are hard to fit into the standard structure of the contextual model. In this paper we discuss how our approaches to recognizing speech acts address the problem of cumulative error. We demonstrate the advantage of the proposed approaches over those that do not address the problem of cumulative error. The experiments are conducted in the context of Enthusiast, a large Spanish-to-English speech-to-speech translation system in the appointment scheduling domain [13, 12, 11, 5]. 1 The Cumulative Error Problem To interpret natural language, it is necessary to take context into account. However, taking context into account can also generate new problems, such as those arising because of cumulative error. Cumulative error is introduced when an incorrect hypothesis is chosen and incorporated into the context, thus providing an inaccurate context from which subsequent context based predictions are made. For example, in Enthusiast, we model the discourse context using speech acts to represent the functions of dialogue utterances [1, 3, 6, 7]. Speech act selection is strongly related to the task of determining how the current input utterance relates to the discourse context. When, for instance, a plan-based discourse processor is used to recognize speech acts, the discourse processor computes a chain of inferences for the current input utterance, and attaches it to the current plan tree. The location of the attachment determines which speech act is assigned to the input utterance. Typically an input utterance can be associated with more than one inference chain, representing different possible speech acts which could be performed by the utterance out of context. Focusing heuristics are used to rank the different inference chains and choose the one which attaches most coherently to the discourse context [3, 8, 5]. However, since heuristics can make wrong predictions, the speech act may be misrecognized, thus making the context inaccurate for future context based predictions. Unexpected input, disfluencies, out of domain utterances, and missing information add to the frequency of misrecognition in spontaneous speech systems. Misrecognition of context states resulting from these features of spontaneous speech adversely affect the quality of contextual information for processing later information. For example, unexpected input can drastically change the standard flow of speechact sequencesin a dialogue. Missing contextual information can make later utterances appear to not fit into the context. 1 Computational Linguistics Program, Carnegie Mellon University, Pittsburgh, PA 15213, USA, email: [email protected] 2 Center for Machine Translation, Carnegie Mellon University, Pittsburgh, PA 15213, USA Cumulative error can be a major problem in natural language systems using contextual information. Our previous experiments conducted in the context of the Enthusiast spontaneous speech translation system show that cumulative error can significantly reduce the usefulness of contextual information [9]. For example, we applied context based predictions from our plan-based discourse processor [7] to the problem of parse disambiguation. Specifically, we combined context based predictions from the discourse processor with non-context based predictions produced by the parser module [4] to disambiguate possibly multiple parses provided by the parser for an input utterance. We evaluated two different methods for combining context based predictions with non-contextbased predictions, namely a genetic programming approach and a neural network approach. We observed that in absence of cumulative error, context based predictions contributed to the task of parse disambiguation. This results in an improvement of 13% with the genetic programming approach and of 2.5% with the neural net approach compared with the parser’s non-context based statistical disambiguation technique.However, cumulative error affected the contribution of contextual information. In the face of cumulative error, the performance decreased by 7.5% for the neural net approach and by 29.5% for the genetic programming approach compared to their respective performances in the absenceof cumulative error, thus dragging the performance statistics of the context based approaches below that of the parser’s non-context based statistical disambiguation technique. The adverse effects of cumulative error in context has been noted in NLP in general. For example, Church and Gale [2] state that “it is important to estimate the context carefully; we have found that poor measures of context are worse than none.” However, we are not aware of this issue having been raised in the discourse processing literature. In the next section, we describe some related work on processing spontaneous dialogues. Section 3 gives a brief description of our system. We discuss the techniques we used to reduce the cumulative error in discourse context for the task of speech act recognition in section 4. Lastly, we evaluate the effects of the proposed approaches on reducing cumulative error.